Neural Architecture Search Algorithms Research Articles

Ancient mural dynasty recognition algorithm based on a neural network architecture search

A neural network model needs to be manually designed for ancient mural dynasty recognition, and this paper proposes an ancient mural dynasty recognition algorithm that is based on a neural architecture search (NAS). First, the structural edge information of mural images is extracted for use by the neural network model in recognizing mural missions. Second, an NAS algorithm that is based on contrast selection (CS) simplifies the architecture search to an incremental CS and then searches for the optimal network architecture on the mural dataset. Finally, the identified optimal network architecture is used for training and testing to complete the mural dynasty recognition task. The results show that the top accuracy of the proposed method on the mural dataset is 88.10%, the recall rate is 87.52%, and the precision rate is 87.69%. Each evaluation index used by the neural network model is superior to that of classical network models such as AlexNet and ResNet-50. Compared with NAS methods such as ASNG and MIGO, the accuracy of mural dynasty recognition is higher by an average of 4.27% when using the proposed method. The proposed method is verified on CIFAR-10, CIFAR-100, ImageNet16-120 and other datasets and achieves a good recognition accuracy in the NAS-bench-201 search space, which averages 93.26%, 70.73% and 45.34%, respectively, on the abovementioned datasets.

DNA Family: Boosting Weight-Sharing NAS With Block-Wise Supervisions.

Neural Architecture Search (NAS), aiming at automatically designing neural architectures by machines, has been considered a key step toward automatic machine learning. One notable NAS branch is the weight-sharing NAS, which significantly improves search efficiency and allows NAS algorithms to run on ordinary computers. Despite receiving high expectations, this category of methods suffers from low search effectiveness. By employing a generalization boundedness tool, we demonstrate that the devil behind this drawback is the untrustworthy architecture rating with the oversized search space of the possible architectures. Addressing this problem, we modularize a large search space into blocks with small search spaces and develop a family of models with the distilling neural architecture (DNA) techniques. These proposed models, namely a DNA family, are capable of resolving multiple dilemmas of the weight-sharing NAS, such as scalability, efficiency, and multi-modal compatibility. Our proposed DNA models can rate all architecture candidates, as opposed to previous works that can only access a sub- search space using heuristic algorithms. Moreover, under a certain computational complexity constraint, our method can seek architectures with different depths and widths. Extensive experimental evaluations show that our models achieve state-of-the-art top-1 accuracy of 78.9% and 83.6% on ImageNet for a mobile convolutional network and a small vision transformer, respectively. Additionally, we provide in-depth empirical analysis and insights into neural architecture ratings.

Parameter-less Pareto local search for multi-objective neural architecture search with the Interleaved Multi-start Scheme

With the emerging deployment of deep neural networks, such as in mobile devices and autonomous cars, there is a growing demand for neural architecture search (NAS) to automatically design powerful network architectures. It is more reasonable to formulate NAS as a multi-objective optimization problem. In addition to prediction performance, multi-objective NAS (MONAS) problems take into account other criteria like the number of parameters and inference latency. Multi-objective evolutionary algorithms (MOEAs) are the preferred approach for tackling MONAS due to their effectiveness in dealing with multi-objective optimization problems. Recently, local search-based NAS algorithms have demonstrated their efficiency over MOEAs for MONAS problems. However, their performance has been only verified on bi-objective NAS problems. In this article, we propose a local search algorithm for multi-objective NAS (LOMONAS), an efficient local search framework for solving not only bi-objective NAS problems but also NAS problems having more than two objectives. We additionally present a parameter-less version of LOMONAS, namely IMS-LOMONAS, by combining LOMONAS with the Interleaved Multi-start Scheme (IMS) to help NAS practitioners avoid manual control parameter settings. Experimental results from a series of benchmark problems in the CEC’23 Competition demonstrate the competitiveness of LOMONAS and IMS-LOMONAS compared to MOEAs in tackling MONAS within both small-scale and large-scale search spaces. Source code is available at: https://github.com/ELO-Lab/IMS-LOMONAS.

A Training-Free Neural Architecture Search Algorithm Based on Search Economics

Motivated by the observation that most neural architecture search (NAS) methods are time-consuming because a “training process” is required to evaluate each searched neural architecture, this paper presents an efficient NAS algorithm based on a promising metaheuristic algorithm named search economics (SE) and a new training-free estimator to evaluate the searched neural architectures for not only obtaining a good neural architecture but also accelerating the computation time. The basic idea of the proposed NAS algorithm is to use the so-called expected value of each region in the search space to guide the search so that it will focus on searching high potential regions instead of solutions with high objective values in particular regions. To evaluate the performance of the proposed algorithm, we compare it with state-of-the-art non-training-free and training-free NAS methods. Experimental results show that the proposed algorithm is capable of finding a result that is similar to or better than those found by most non-training-free NAS algorithms compared in this study but taking only a tiny portion of the computation time.

Bandit-NAS: Bandit sampling and training method for Neural Architecture Search

Existing Neural Architecture Search algorithms achieve a low error rate in vision tasks, such as image classification, by training child networks with equal resources during the search. However, it is unnecessary to allocate equal resources or fully converge scores to assess which child architectures should be adopted, resulting in computational redundancy. In this study, we present Bandit-NAS, an approach that automatically computes data slicing and training time for each child network. Firstly, we formulate the search for the optimal training time for a given resource as an M-armed bandit problem. Secondly, we extend the original NAS methods by proposing an end-to-end bandit algorithm, combined with reinforcement learning-based NAS algorithms, to determine an update strategy. Bandit-NAS enables simultaneous training of M child networks within a specified resource constraint (one epoch training time), with the allocation of training data based on the current accuracy of the child networks, thereby minimizing their error rate. Experimental results on 3 different datasets, MNIST, CIFAR-10 and CIFAR-100 demonstrate the superiority of Bandit-NAS over baseline NAS algorithms, such as ENAS and DQNAS, achieving lower error rates and faster search time.

Architecture Augmentation for Performance Predictor via Graph Isomorphism.

Neural architecture search (NAS) can automatically design architectures for deep neural networks (DNNs) and has become one of the hottest research topics in the current machine learning community. However, NAS is often computationally expensive because a large number of DNNs require to be trained for obtaining performance during the search process. Performance predictors can greatly alleviate the prohibitive cost of NAS by directly predicting the performance of DNNs. However, building satisfactory performance predictors highly depends on enough trained DNN architectures, which are difficult to obtain due to the high computational cost. To solve this critical issue, we propose an effective DNN architecture augmentation method named graph isomorphism-based architecture augmentation method (GIAug) in this article. Specifically, we first propose a mechanism based on graph isomorphism, which has the merit of efficiently generating a factorial of n (i.e., n) diverse annotated architectures upon a single architecture having n nodes. In addition, we also design a generic method to encode the architectures into the form suitable to most prediction models. As a result, GIAug can be flexibly utilized by various existing performance predictors-based NAS algorithms. We perform extensive experiments on CIFAR-10 and ImageNet benchmark datasets on small-, medium-and large-scale search space. The experiments show that GIAug can significantly enhance the performance of the state-of-the-art peer predictors. In addition, GIAug can save three magnitude order of computation cost at most on ImageNet yet with similar performance when compared with state-of-the-art NAS algorithms.

Resource-Aware Federated Neural Architecture Search over Heterogeneous Mobile Devices

Federated learning has been recently proposed for many clients to collaboratively train a machine learning model in a privacy-preserving manner. However, it also amplifies the difficulty in designing good neural network architecture, especially considering heterogeneous mobile devices. To this end, we propose a novel neural architecture search algorithm, namely FedNAS, which can automatically generate a set of optimal models under federated settings. The main idea is to decouple the two primary steps of the NAS process, i.e, model search and model training, and separately distribute them on the cloud and devices. It also tackles the primary challenge of limited on-device computational and communication resources through its novel designs: FedNAS fully exploits the key opportunity of insufficient model candidate re-training during the architecture search process and incorporates three key optimizations: parallel candidate training on partial clients, early dropping candidates with inferior performance, and dynamic round numbers. Evaluated on typical CNN architectures and large-scale datasets, FedNAS is able to achieve comparable model accuracy as a state-of-the-art NAS algorithm that trains models with centralized data, and also reduces the client cost by up to 200_ or more compared to a straightforward design of federated NAS.

A Fast Evolutionary Knowledge Transfer Search for Multiscale Deep Neural Architecture.

The emergence of neural architecture search (NAS) algorithms has removed the constraints on manually designed neural network architectures, so that neural network development no longer requires extensive professional knowledge, trial and error. However, the extremely high computational cost limits the development of NAS algorithms. In this article, in order to reduce computational costs and to improve the efficiency and effectiveness of evolutionary NAS (ENAS) is investigated. In this article, we present a fast ENAS framework for multiscale convolutional networks based on evolutionary knowledge transfer search (EKTS). This framework is novel, in that it combines global optimization methods with local optimization methods for search, and searches a multiscale network architecture. In this article, evolutionary computation is used as a global optimization algorithm with high robustness and wide applicability for searching neural architectures. At the same time, for fast search, we combine knowledge transfer and local fast learning to improve the search speed. In addition, we explore a multiscale gray-box structure. This gray box structure combines the Bandelet transform with convolution to improve network approximation, learning, and generalization. Finally, we compare the architectures with more than 40 different neural architectures, and the results confirmed its effectiveness.

Particle Swarm Optimization for Efficiently Evolving Deep Convolutional Neural Networks Using an Autoencoder-based Encoding Strategy

Deep convolutional neural networks (DCNNs) have achieved surpassing success in the field of computer vision, and a number of elaborately designed networks refresh the performance records in benchmark datasets. Recently, evolutionary neural architecture search (ENAS) has become an emerging area, which could employ an evolutionary computation (EC) technique to automatically construct promising network architectures without human intervention. However, existing algorithms still have limitations: most standard EC approaches cannot process flexible-length architecture representations directly, and most fitness evaluation processes suffer from the exhibitive computational cost and the unreliable prediction results. To overcome these shortcomings, we propose an efficient particle swarm optimization (PSO)-based neural architecture search algorithm to search for appropriate dense blocks on image classification tasks, named EAEPSO. EAEPSO addresses the first limitation by designing an autoencoder to encode variable-length network representations as fixed-length latent vectors, which converts the original search space to a latent space that can facilitate the downstream search. Besides, an efficient and effective hierarchical fitness evaluation method is designed to guide the search process to address the second limitation. The experimental results show that EAEPSO is a very competitive ENAS algorithm that achieves an error rate of 2.74% on CIFAR-10 and 16.17% on CIFAR-100, and reduces the computational cost from hundreds or thousands of GPU-days to only 2.2 and 4 GPU-days, respectively. Further analyses investigate the reduced training data’s effect and confirm the effectiveness of both the proposed autoencoder and the proposed hierarchical fitness evaluation method.

NDARTS: A Differentiable Architecture Search Based on the Neumann Series

Neural architecture search (NAS) has shown great potential in discovering powerful and flexible network models, becoming an important branch of automatic machine learning (AutoML). Although search methods based on reinforcement learning and evolutionary algorithms can find high-performance architectures, these search methods typically require hundreds of GPU days. Unlike searching in a discrete search space based on reinforcement learning and evolutionary algorithms, the differentiable neural architecture search (DARTS) continuously relaxes the search space, allowing for optimization using gradient-based methods. Based on DARTS, we propose NDARTS in this article. The new algorithm uses the Implicit Function Theorem and the Neumann series to approximate the hyper-gradient, which obtains better results than DARTS. In the simulation experiment, an ablation experiment was carried out to study the influence of the different parameters on the NDARTS algorithm and to determine the optimal weight, then the best performance of the NDARTS algorithm was searched for in the DARTS search space and the NAS-BENCH-201 search space. Compared with other NAS algorithms, the results showed that NDARTS achieved excellent results on the CIFAR-10, CIFAR-100, and ImageNet datasets, and was an effective neural architecture search algorithm.

NPENAS: Neural Predictor Guided Evolution for Neural Architecture Search.

Neural architecture search (NAS) adopts a search strategy to explore the predefined search space to find superior architecture with the minimum searching costs. Bayesian optimization (BO) and evolutionary algorithms (EA) are two commonly used search strategies, but they suffer from being computationally expensive, challenging to implement, and exhibiting inefficient exploration ability. In this article, we propose a neural predictor guided EA to enhance the exploration ability of EA for NAS (NPENAS) and design two kinds of neural predictors. The first predictor is a BO acquisition function for which we design a graph-based uncertainty estimation network as the surrogate model. The second predictor is a graph-based neural network that directly predicts the performance of the input neural architecture. The NPENAS using the two neural predictors are denoted as NPENAS-BO and NPENAS-NP, respectively. In addition, we introduce a new random architecture sampling method to overcome the drawbacks of the existing sampling method. Experimental results on five NAS search spaces indicate that NPENAS-BO and NPENAS-NP outperform most existing NAS algorithms, with NPENAS-NP achieving state-of-the-art performance on four of the five search spaces.

Problems of Neural Network Architecture Benchmarking and Search

In this paper we made a survey of neural architecture search algorithms and their benchmarking. Based on our survey we highlight the current problems in the quality of neural network architecture benchmarking and in the comparison of neural architecture search algorithms.

HGNAS++: Efficient Architecture Search for Heterogeneous Graph Neural Networks

Heterogeneous graphs are commonly used to describe networked data with multiple types of nodes and edges. Heterogeneous Graph Neural Networks (HGNNs) are powerful tools for analyzing heterogeneous graphs. However, designing neural architectures of HGNNs requires extensive domain knowledge and time-consuming manual work. Recently, neural architecture search algorithms have become popular in automatically designing neural architectures for homogeneous graph neural networks. In this paper, we present a Heterogeneous Graph Neural Architecture Search algorithm (HGNAS for short) which allows the automatic design of heterogeneous graph neural architectures. Specifically, HGNAS first designs a new search space based on existing popular HGNNs. Then, HGNAS uses a policy network as the controller to sample and find the best neural architecture from the designed search space by maximizing the expected accuracy of the selected architectures on a given validation dataset. Moreover, we design a new method HGNAS++ to improve the efficiency of HGNAS by training the RNN controller within a generative adversarial learning framework. The basic idea of HGNAS++ is to embed a pairwise ranker into the reinforcement learning based architecture search algorithm. The pairwise ranker can be taken as a discriminator which selects more accurate architectures between pairs of candidate architectures. Then, the RNN controller can be updated more efficiently by only using a relatively small number of candidate architectures selected by the pairwise ranker. Experiments on real-world heterogeneous graph datasets show that HGNAS is capable of designing novel HGNNs that beat the best human-invented HGNNs. On the benchmark datasets, HGNAS++ improves HGNAS in terms of evaluation cost, with a reduction of 50% of the evaluated candidate architectures and a decrease of 24% in search time on average. As a byproduct, HGNAS++ can find sparse yet powerful neural architectures for HGNNs.

NG-NAS: Node growth neural architecture search for 3D medical image segmentation.

Neural architecture search (NAS) has been applied to design proper 3D networks for medical image segmentation. In order to reduce the computation cost in NAS, researchers tend to adopt weight sharing mechanism to search architectures in a supernet. However, recent studies state that the searched architecture rankings may not be accurate with weight sharing mechanism because the training situations are inconsistent between the searching and training phases. In addition, some NAS algorithms design inflexible supernets that only search operators in a pre-defined backbone and ignore the importance of network topology, which limits the performance of searched architecture. To avoid weight sharing mechanism which may lead to inaccurate results and to comprehensively search network topology and operators, we propose a novel NAS algorithm called NG-NAS. Following the previous studies, we consider the segmentation network as a U-shape structure composed of a set of nodes. Instead of searching from the supernet with a limited search space, our NG-NAS starts from a simple architecture with only 5 nodes, and greedily grows the best candidate node until meeting the constraint. We design 2 kinds of node generations to form various network topological structures and prepare 4 candidate operators for each node. To efficiently evaluate candidate node generations, we use NAS without training strategies. We evaluate our method on several public 3D medical image segmentation benchmarks and achieve state-of-the-art performance, demonstrating the effectiveness of the searched architecture and our NG-NAS. Concretely, our method achieves an average Dice score of 85.11 on MSD liver, 65.70 on MSD brain, and 87.59 in BTCV, which performs much better than the previous SOTA methods.

A reinforcement neural architecture search convolutional neural network for rolling bearing fault diagnosis

The complexity of machinery makes accurate identification of rolling bearing fault signals difficult. Convolutional neural networks (CNNs) have made some progress, but they rely on the expertise of the network designer and the iterative process of optimizing numerous parameters. Therefore, there is an urgent need to develop a method that reduces the threshold for designing CNNs for a given task. In this article, we propose a reinforcement neural architecture search CNN to address this problem. Firstly, we design a neural architecture search algorithm that can generate different types of sub-networks specifically for fault diagnosis tasks. Secondly, we execute a reinforcement learning-based search strategy to discover promising sub-networks. Furthermore, we enhance the performance of the sub-network by improving the optimizer and training parameters. We conduct extensive experiments using two different types of datasets and verify that the proposed method’s fault classification capability is superior to existing methods.

ANAS: Asymptotic NAS for large-scale proxyless search and multi-task transfer learning

Neural Architecture Search (NAS) is an emerging solution to design a lightweight network for researchers to obtain a trade-off between accuracy and speed, releasing researchers from tedious repeated trials. However, the main shortcoming of NAS is its high and unstable memory consumption of the search work, especially for large-scale tasks. In this study, the proposed Asymptotic Neural Architecture Search network (ANAS) achieved a proxyless search for large-scale tasks with economic and stable memory consumption. Instead of proxy search like other NAS algorithms, ANAS achieved the large-scale proxyless search that directly learns deep neural network architecture for target task. ANAS reduced the peak value of memory consumption by an asymptotic method, and kept the memory consumption stable by the linkage change of a series of key indexes. A pruning operation and efficient candidate operations decreased the total memory consumption. Finally, ANAS achieved a good trade-off between accuracy and speed for classification tasks on CIFAR-10, CIFAR-100, and ImageNet datasets. Besides, except for the classification task, it achieved excellent multi-task transfer learning ability for implementing the segmentation task on CamVid and Cityscapes. ANAS reached 22.8% test errs with 5 M parameter on ImageNet, and 72.9 mIoU (mean Intersection over Union) with 119.9 FPS (Frames Per Second) on Cityscapes dataset.

Neural Architecture Search Based on a Multi-Objective Evolutionary Algorithm With Probability Stack

With the emergence of deep neural networks, many research fields, such as image classification, object detection, speech recognition, natural language processing, machine translation and automatic driving, have made major breakthroughs in technology and the research achievements have been successfully applied in many real-life applications. Combining evolutionary computation and neural architecture search (NAS) is an important approach to improve the performance of deep neural networks. Usually, the related researchers only focus on precision. Thus, the searched neural architectures always perform poorly in the other indexes such as time cost. In this paper, a multi-objective evolutionary algorithm with a probability stack (MOEA-PS) is proposed for NAS, which considers the two objects of precision and time consumption. MOEA-PS uses an adjacency list to represent the internal structure of deep neural networks. Besides, a unique mechanism is introduced into the multi-objective genetic algorithm to guide the process of crossover and mutation when generating offspring. Furthermore, the structure blocks are stacked using a proxy model to generate deep neural networks. The results of the experiments on Cifar-10 and Cifar-100 demonstrate that the proposed algorithm has a similar error rate compared with the most advanced NAS algorithms, but the time cost is lower. Finally, the network structure searched on Cifar-10 is transferred directly to the ImageNet dataset, which can achieve 73.6% classification accuracy.

ATNAS: Automatic Termination for Neural Architecture Search

Neural architecture search (NAS) is a framework for automating the design process of a neural network structure. While the recent one-shot approaches have reduced the search cost, there still exists an inherent trade-off between cost and performance. It is important to appropriately stop the search and further reduce the high cost of NAS. Meanwhile, the differentiable architecture search (DARTS), a typical one-shot approach, is known to suffer from overfitting. Heuristic early-stopping strategies have been proposed to overcome such performance degradation. In this paper, we propose a more versatile and principled early-stopping criterion on the basis of the evaluation of a gap between expectation values of generalisation errors of the previous and current search steps with respect to the architecture parameters. The stopping threshold is automatically determined at each search epoch without cost. In numerical experiments, we demonstrate the effectiveness of the proposed method. We stop the one-shot NAS algorithms and evaluate the acquired architectures on the benchmark datasets: NAS-Bench-201 and NATS-Bench. Our algorithm is shown to reduce the cost of the search process while maintaining a high performance.

POPNASv3: A pareto-optimal neural architecture search solution for image and time series classification

The growing demand for machine learning applications in industry has created a need for fast and efficient methods to develop accurate machine learning models. Automated Machine Learning (AutoML) algorithms have emerged as a promising solution to this problem, designing models without the need for human expertise. Given the effectiveness of neural network models, Neural Architecture Search (NAS) specialises in designing their architectures autonomously, with results that rival the most advanced hand-crafted models. However, this approach requires significant computational resources and hardware investment, making it less attractive for real-world applications. This article presents the third version of Pareto-Optimal Progressive Neural Architecture Search (POPNASv3), a new NAS algorithm that employs Sequential Model-Based Optimisation and Pareto optimality. This choice makes POPNASv3 flexible to different hardware environments, computational budgets and tasks, as the algorithm can efficiently explore user-defined search spaces of varying complexity. Pareto optimality extracts the architectures that achieve the best trade-off with respect to the metrics considered, reducing the number of models sampled during the search and dramatically improving time efficiency without sacrificing accuracy. The experiments performed on image and time series classification datasets provide evidence that POPNASv3 can explore a large set of different operators and converge to optimal architectures suited to the type of data provided under different scenarios.11The algorithm is open source and available on Zenodo at https://zenodo.org/record/7439022. The repository contains the code, the URIs of the datasets used in the experiments, and the models stored in ONNX format.

A surrogate evolutionary neural architecture search algorithm for graph neural networks

Due to the unique construction module and design of graph neural networks (GNNs), neural architecture search (NAS) methods specifically for GNNs have become a promising research hotspot in recent years. Among the existing methods, one class of methods microscopically searches for the constituent components of network layers. However, most of them ignore the topology connections between network layers or the feature fusion strategies. Another class of methods, called differentiable architecture search methods, has the advantage of searching topology connections and feature fusion strategies. However, constrained by the requirement of predefining all candidate operations, these methods can only sample a limited number of network layers. In this paper, we propose a surrogate evolutionary graph neural architecture search (GNAS) algorithm whose search space contains not only the microscopic network layer components but also topology connections and feature fusion strategies (called CTFGNAS). The GNN sampled in CTFGNAS is represented by a simple one-dimensional vector and does not fix the network depth. To address the problem that traditional crossover and mutation operators applied to GNAS may produce illegal solutions, we design a repair operation to guarantee the legitimacy of the solutions. The network depth is also increased with a large probability in the mutation operation to alleviate the oversmoothing problem. In addition, to cope with the challenge of computational resources due to the increased search space, we form a surrogate model with three classical regression models, where only a small number of solutions are truly evaluated for their fitness, and the remaining large number of solutions are predicted for their fitness by the surrogate model. Finally, experiments are executed on six widely used real-world datasets. The experimental results illustrate that CTFGNAS obtains more effective results than the state-of-the-art handcrafted GNNs and GNAS methods on all datasets. CTFGNAS is now available on the following website: https://github.com/chnyliu/CTFGNAS.